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Abstract 

In this paper we find and classify all patterns for a single locus three- and four-allele 
population genetics models in continuous time. A pattern for a fc-allele model means 
all coexisting locally stable equilibria with respect to the flow defined by the equations 
Pi = Pi( r i ~ r )ii — 1, - ■ ■ ,k, where Pi,ri are the frequency and marginal fitness of allele 
Ai, respectively, and r is the mean fitness of the population. It is well known that for the 
two-allele model there are only three patterns depending on the relative fitness between 
the homozygotes and the heterozygote. It turns out that for the three-allele model there 
are 14 patterns and for the four-allele model there are 117 patterns. With the help of 
computer simulations, we find 2351 patterns for the five- allele model. For the six-allele 
model, there are more than 60, 000 patterns. In addition, for each pattern of the three- 
allele model, we also determine the asymptotic behavior of solutions of the above system 
of equations as t — > oo. The problem of finding patterns has been studied in the past 
and it is an important problem because the results can be used to predict the long-term 
genetic makeup of a population. 

Key words: k-allele model, stable equilibrium, patterns, computer simulations. 

1 Introduction 



Selection is an important driving force in evolution and is due mainly to the differences in fitness 
to survival between genotypes in a population. Since the 1920 's many mathematical models for 
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selection have been developed and studied and many results are known [13j E] • In this paper 
we give a partial answer to an old but still open problem mentioned in [3j p. 38] concerning the 
maximum possible number of coexisting stable equilibria in the evolution of gene frequencies 
under selection forces in a single-locus multiple alleles population genetics model. 

Consider a diploid population whose members possess a gene that occurs in k different forms, 
called alleles, located at an autosomal locus. Let the fitness of genotype AiAj be denoted by 
Tij and R = (r^) is the k by k symmetric fitness matrix. Let pi be the frequency of allele A\. 
Then, assuming random mating and discrete-time non-overlapping generations, the frequency 
of allele A4 in the next generation is given by 

p'i^PiTi/r, i = l,...,k, 

where r, = ^ . r^pj is the marginal fitness of allele A4 and r = £\ r^j is the mean fitness of 
the population. For continuous-time overlapping generations, above equations are replaced by 

Pi=Pi(n-r), i = l,...,k, (1.1) 

which is only an approximation, derived under the assumption of Hardy- Weinberg equilibrium 
0, Section 10.1]. In this paper, we focus our study on the continuous-time model (11. II) . Our 
main idea is to determine the signs of eigenvalues of the Jacobian matrix evaluated at an existing 
equilibrium to determine if the equilibrium is stable or not. The definitions of mathematical 
terms used in this paper will be given in Section El 

Let Pi(t),i = 1, . . . , k be the solutions of (11.11) . Since they are the frequencies at the same 
locus, they must add up to one; that is ^2Pi(i) = 1. Therefore, (11.11) is actually a system of 
(k — 1) equations with p k replaced by (1 — pi — ■ • • — Pk-i)- By relabeling the alleles, we may 
assume that r&'s are non- increasing in % but for simplicity, we assume that they are decreasing; 
that is, 

r u > r 22 > ■ ■ ■ > r kk . (1.2) 

This paper is organized as follows. In Section El we present the mathematical background 
necessary to understand this paper. In Section [3l we present the well known results of the 
two-allele model, define what is a pattern, and review the literature on the subject of finding 
patterns. In Section HJ we explain how one can obtain the eigenvalues of the Jacobian matrix 
evaluated at an equilibrium. In Section [51 we give a rigorous proof that there are 14 patterns 
for the three-allele model. A complete description of the dynamics of the solutions of ( ll.ip is 
also included. In Section [61 we present our computer simulations that led to the discovery of 
patterns for the four- and five-allele models. Section [7] is conclusion. In order to make this 
paper accessible to a wider audience, we omit most of the proofs and instead focus on examples 
to illustrate our results. 
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2 Mathematical Background 



In this section, we give a brief introduction to the definitions of the mathematical terms used 
in this paper. Further details may be found in the book [7]. Let 

fi(pi,...,Pk-i)=Pi(ri-r), i = 1, (2.1) 

where rj,r are defined in section [I] with pk = 1 — p\ — ■ ■ ■ — Pk-i- An equilibrium solution, 
P* = (p*j ■ • • iPk-\)i i s a constant solution of the nonlinear system of equations (12. ip . The 
Jacobian matrix of /j's is 

9 ft 

J = (Jij) where the i th row j th column component is Jy = . 

If we want to find the behavior of the solutions of (11. ip as time goes to infinity (t — > oo) with 
a starting point near an equilibrium p*, one can instead study the linear system 

p = J*p, (2.2) 

where J* is the constant matrix obtained by evaluating J at the equilibrium p*. For further 
details and examples see [7J Section 5.7]. 

The behavior of solutions to (12. 2p depends on the signs of the eigenvalues of J*. A constant 
A, which may be a complex number, is an eigenvalue of J* if there exists a nonzero k — 1- 
dimensional vector, x, such that J*x = Ax. It is well known that J* has k — 1 eigenvalues and 
p* is asymptotically stable if and only if all the eigenvalues of J* have negative real-parts. If p* is 
asymptotically stable, then there exists a small ball centered at p* (also called a neighborhood) 
such that the solutions of (II. ip with initial value inside this ball converge to p* as t — > oo. In 
this paper, for simplicity, we say p* is stable instead of asymptotically stable. This should not 
cause any confusion. It turns out that for the population genetics model, the eigenvalues of J* 
is always real and a large part of section [4] is to show how to find the signs of the eigenvalues 
of J*. There are some linear algebra terms needed to understand this paper. Details may be 
found in the book [15J. Determinant of J* is the product of all the eigenvalues of J* and the 
trace of J* is the sum of all the eigenvalues of J*. If J* is of size 2 (three-allele case), let Ai, A2 
be the eigenvalues of J*, then det( J) = AiA 2 and tr( J) = Ai + A 2 . Therefore, we can determine 
the signs of the eigenvalues from the signs of det(J*) and tr(J*). This fact will be used in 
Section [51 Let A be a square matrix. Choose any entry aij of A that lies in the i th -th row and 
j'^-column of A. Cross out the i th -row and j^-column of A and denote the determinant of the 
resulting matrix by My. Then (— l) 4+1 Mjj is called the cofactor of and M = (My) is called 
the cofactor matrix of A. Cofactors will be used in Section HI The transpose of the cofactor 
matrix M is called the adjoint of A. A permutation matrix is a matrix containing only l's and 
O's with each row and column containing exactly one 1. Multiplying A on the left and right by 
the same permutation matrix will interchange certain rows and columns A. 
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Let us return to the study of our nonlinear system (11. ip . Since the frequencies of the alleles 
always lie between and 1 and add up to one, the solutions of (jl.ip always lie inside the 

invariant set 



k-l 



A = {(pi, . . .,pk-i) ■ < pi < 1 for 



% = 1, < 1}. 



(2.3) 



i=l 



Invariant here means solutions of (11.11) that start inside A remain in A for all future time. 
Also, because A is invariant, all equilibria of (11. II) lie inside A. A boundary equilibrium, p*, is 
an equilibrium where at least one of the components of p* is zero. An interior equilibrium is 
one where all the components are positive. It is known that solutions to (II. ip always converge 
to a stable equilibrium as t — > oo |5J p. 38]. If there are more than one stable equilibrium in 
A, then A can be partitioned into non-overlapping regions by hypersurfaces (which are curves 
when k — 3) called separatrices. Each of these regions contains one stable equilibrium and 
solutions starting in a region will converge to the stable equilibrium in that region as t — > oo. 
As an example, consider the first figure in Figure [1] in the proof of Theorem 15.51 The vertices 
(1, 0) and (0, 1) are the only stable equilibria. The separatrix is the curve shown joining the 
origin and P3 to the interior equilibrium. Solutions that start on the separatrix converge to 
the interior equilibrium while solutions that start above or below it converge to (0, 1) or (1, 0), 
respectively as t — > 00. 

3 Two-allele Model and Finding All Possible Patterns 

Let us consider the simplest case with two alleles. Since p\ + p 2 = 1, A = [0, 1] and system 
(II. ip may be reduced to the single equation 



where A12 = rn + r 22 — 2ri 2 . The following result is well known [T3] . 

Proposition 3.1. For the two-allele model ( Iff, ij) with rn > r 22 , Pi = and p\ = 1 are both 
equilibrium and a third interior equilibrium p* = (r 22 — r 12 )/A 12 exists if and only ifr i2 > r±i or 
r u < r 22- There are three cases: (i) r 22 < r 12 < r n : 1 is stable, is unstable and p* does not 
exist; (ii) r 12 > r±i : 0, 1 are both unstable and p* exists and is stable; (Hi) r 12 < r 22 : 0, 1 are 
both stable, p* exists and is unstable. We call theses three cases the (hetero zygote) intermediate, 
superior, and inferior cases, respectively. If one of the alleles is completely dominant; i.e. 
^12 = fi\ or r\2 = 1*22, then p* does not exist and Ii3. 1\) reduces to the intermediate case with a 
double zero either at p\ = or at p\ — 1. 

The above proposition implies that for the two-allele model there are only three patterns 
depending on the values of ri 2 relative to rn and r 22 . Pattern 1 is when 1 is the only stable 
equilibrium, pattern 2 is when p* is the only stable equilibrium, and pattern 3 is when and 1 are 




(3.1) 
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both stable and the interior equilibrium p* is unstable. In Pattern 3, and 1 are coexisting stable 
equilibria. We denote these three patterns by {1}, {p*}, and {0,1}, respectively. From this 
example, we shall define pattern as the number and locations of all coexisting stable equilibria. 
Note that {1} and {p*} are different patterns because 1 is a boundary equilibrium but p* is 
an interior equilibrium. However, even though p* varies depending on the value of ri2, we still 
consider {p*} as one pattern because it lies in the interior. For the fc-allele model there are 
(2 fc — 1) possible equilibria but the number of coexisting stable equilibria in any pattern is 
considerably less. For example, for the five-allele model there are 2 5 — 1 = 31 possible equilibria 
but no pattern can contain more than 6 stable equilibria. 

The problem of finding all patterns for a /c-allele model has been studied extensively. In a 
series of papers (2j HE! @J El E], the authors studied maximal patterns, that is, patterns which 
are not a subset of another pattern, for 3,4,5 alleles. Cannings and Vickers conjectured that 
any subset of a pattern is also a pattern (T6J Ej. Thus, if this conjecture is true, one may specify 
the complete set of patterns by specifying the complete set of maximal patterns. This is the 
approach adopted in the literature. In this paper, we give a complete list of patterns for the 
case k = 3 (proved analytically), k = 4 (proved and confirmed by computer simulations), and 
k = 5 (with a few unsolved cases) without assuming that the conjecture is true. Note also 
that (11. 2p is not posited in the literature. Therefore, patterns that are rotations of each other 
can be obtained by multiplying the fitness matrix R on both sides by a permutation matrix 
and thus are considered as the same pattern. However, under condition (II. 2p . rotation of a 
pattern is not necessarily a pattern. For example, for the three-allele model, A is the triangle 
with vertices at (1, 0), (0, 1) and (0, 0). The vertex {(1, 0)} is a pattern but the vertices {(0, 1)} 
and {(0,0)} are not patterns even though they can be obtained from {(1,0)} by rotation. 
Therefore, we treat patterns that are rotations of each other as different patterns. For this 
reason, the number of patterns we obtained in this paper is greater than those obtained in the 
literature. The method we used in this paper, counting signs of eigenvalues of the Jacobian 
matrix evaluated at the existing equilibria, is also completely different from the method used 
in the literature. Furthermore, we also consider the case of complete dominance and describe 
all possible asymptotic behavior of the solutions of (II. ip when k = 3. 



4 Properties of System (11.11) 



We begin with the problem of existence of interior equilibria. Let Rij be the cofactor of r^ in 
the fitness matrix R and let U{ = Ylj Rij- The following result is due to Mandel |llj . 

Proposition 4.1. Suppose detR 7^ 0. An interior equilibrium exists if and only if all Ui 's are 
non-zero and have the same sign. Moreover, if the interior equilibrium exists, it is given by 

v--^- .-1 k 

Z^j U 3 
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The degenerate case det R = is considered in [TJ] and [TJ. In this paper, we always assume 
that the determinant of R and all its submatrices are non-zero. A submatrix of R is the matrix 
obtained by erasing any number of rows and the corresponding columns of R. 

Next we consider the stability of equilibria of system ( II. ip . It is well known that the mean 
fitness r, defined in Section HJ is strictly increasing along non-constant solutions of ( 11. ip [10J. 
Therefore, r is what is commonly called a Lyapunov function and the stable equilibria are 
identified with the local maxima of r. Quite a few necessary and sufficient conditions for the 
stability of an existing equilibrium can be obtained by studying the properties of local maxima 
of r, see [121 E] an d the references therein. Another more direct approach is to investigate 
the eigenvalues of the Jacobian matrix J evaluated at an equilibrium of system ( II. ip . It can 
be shown that all such eigenvalues are real. A proof of this fact for the discrete-time model is 
given in Section 3]. We adopt the second more direct approach in this paper. 

Let p* = (p*, . . . ,pl) be an equilibrium whose components add up to one. Let K be the 
set {1, . . . , k} and define L = {i 6 K : p* > 0} and L = {i G K : p* — 0}. The following 
proposition gives the formula of the Jacobian matrix J which will be used extensively in this 
paper. The result is probably known but we are unable to find a reference for it so we present 
a proof here. Readers not interested in the proof may simply read Remark 14.31 to understand 
how this proposition is used. 

Proposition 4.2. Let I e K and replace pi by (1 — X^y-P*) ^ n (E2P- Then the i th -row, j th - 
column of the Jacobian matrix J of ( II. ip (as a system of (k — 1) equations) evaluated at an 
equilibrium point p* is given by 




i,j = l,...,fc- 1, 




In particularly, if p* is an interior equilibrium so that L — 0, then 

J ij = Pi( r ij ~ r li) > 2, j = 1, 1 . 





2_^Pt) ■ Therefore, 




n - 



r-jiPi + nipi + r- + r fj pi - r u pi - n 



2(r~. - rj). 
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Noting that r, = r for i G L and at an interior equilibrium, n = 
proposition follows. 



= rfc, the conclusion of the 

□ 



It can also be shown that at an interior equilibrium p* the determinant of the Jacobian 
matrix J is given by 

det(J) = plpl ■ ■ -pl{U x + U 2 + ■ ■ ■ + U k ). (4.1) 
Remark 4.3. If we choose I to be in L, then 

13 I M^-OIp- if 

This implies that the eigenvalues of J consist of (r* — r)| p * (i G L) and the eigenvalues of the 
matrix (p*(rij — rj/))j Jgi _{;}. This formula greatly simplifies computation of the eigenvalues 
of J. For example, for a five-allele model, if we want to find the eigenvalues of J at the 
boundary equilibrium on the side where p 2 = 0,^5 = 0, all we have to do is find the interior 
equilibrium of the three-allele model with fitness matrix obtained by eliminating the second 
and fifth rows and columns of R. Let this interior equilibrium be denoted by {pi,p2,p3) and let 
p* = (pi, 0,p2,p3, 0). Then according to the above proposition with / = 4, the eigenvalues of J 
evaluated at p* consists of the eigenvalues of the matrix 

( pi(ru-ru) pi(r 13 -r 14 )\ 

V P2O31 - ^34) P2O33 - ^34) / 

and Vi ■ p* — Rp* ■ p*, i = 2, 5 where is the i th -row of the matrix R. 

Remark 4.4. From Section [21 an equilibrium is stable if all the eigenvalues of J at that 
equilibrium have negative real parts and unstable if at least one eigenvalue has positive real 
part. Moreover, Equation (14) in [5] shows that zero eigenvalues do not affect the stability 
of a boundary equilibrium of system (11. ip . This together with formula (14.11) imply that an 
equilibrium of system (II. ip is stable if and only if all the eigenvalues are non-positive. 

The following result will be used later in our study. The proof for the discrete-time case 
may be found in [9]. 

Proposition 4.5. Suppose A x C A 2 are two distinct invariant subsets of A and the corre- 
sponding interior equilibria exist, then they cannot be both stable. In particular, if A contains 
a stable interior equilibrium, then no other equilibrium in A can be stable. 

For example, for the four-allele case, A is a tetrahedron. Let A 2 be one of the four faces 
of A and suppose it has a stable interior equilibrium. Then Proposition 14.51 says that none 
of the vertices or the edges of A 2 (either can be Ai in Proposition I4.5P can contain a stable 
equilibrium. 
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5 Three-allele Model 



The invariant set A defined by ( 12. 3ft is the triangle joining the vertices (1, 0), (0, 1) and (0, 0). 
System ( 11. ip is 

Pi=Pi{r i -r) 1 i = l,2. (5.1) 

We denote the three sides of A by Si = {(pi,P2) '■ Pi = 0},z = 1,2,3 and the boundary equi- 
librium on Si by Pi if it exists. Each side is a two- allele model so according to Proposition 13.11 
Pi exists if Si is in the superior case or inferior case and Pi is stable and unstable, respectively. 
We now introduce some terminologies. Consider alleles Ai and Aj. If = ra, we say that 
Ai is completely dominant to Aj and Aj is recessive to A^. Partial dominance means lies 
between ra and Tjj but not equal to their average and no dominance means = {ja + rjj)/2. 
For convenience, we say that Si is completely dominant if r23 = r^2 or r23 = holds. Likewise 
for edges S2 and S3. 

5.1 Non- Complete Dominance Case 

In this subsection we assume that 7^ ra,rjj. There are altogether 14 patterns. Five of them 
have one stable equilibrium, eight have two stable equilibria, and one has three stable equilibria. 
It is difficult to prove these results directly so we state four lemmas according to how many 
Pi's exist on the boundary of A and whether they are in the intermediate, superior or inferior 
cases. The classifications of the 14 patterns will then follow easily from these lemmas and are 
summarized in Tables [1] - [3j For simplicity we assume that all the eigenvalues are nonzero (see 
Remark I4.4p . The proofs of the four lemmas are rather technical and are given at the end of 
this subsection. Readers not interested in proof may simply look at the tables. 

Lemma 5.1. Suppose none of the Pi's exists. Then the interior equilibrium does not exist. 

Lemma 5.2. Suppose only one Pi exists. Then there are 8 cases: 

1. If P2 exists but not Pi,Ps, then there are two cases: (i) S2 is in the inferior case, P2 has 
two positive eigenvalues and interior equilibrium does not exist, (ii) S2 is in the superior 
case, P2 has two negative eigenvalues and interior equilibrium does not exist. 

2. If Pi exists but not P2, P3, then there are three cases: (i) Si is in the inferior case, Pi has 
two positive eigenvalues and interior equilibrium does not exist, (ii) Si is in the superior 
case, Pi has two negative eigenvalues and interior equilibrium exists with one positive and 
one negative eigenvalues, (Hi) Si is in the superior case, Pi has one negative and one 
positive eigenvalues and interior equilibrium does not exist. 

3. If P3 exists but not Pi, P2, then there are three cases: (i) S3 is in the superior case, P3 has 
two negative eigenvalues and interior equilibrium does not exist, (ii) S3 is in the inferior 
case, P3 has two positive eigenvalues and interior equilibrium exists with one positive 
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and one negative eigenvalue, (Hi) S3 is in the inferior case, P3 has one positive and one 
negative eigenvalues and interior equilibrium does not exist. 

Lemma 5.3. Suppose only two Pi 's exist. Then there are 34 cases: 

A. Suppose P21P3 exist but not P 1 . Then there are 11 cases: 

1. If S*2, S*3 are both in the superior case, then there are 4 cases. 

(a) P 2 or P3 is stable with two negative eigenvalues and the other is unstable with one 
positive and one negative eigenvalues. Interior equilibrium does not exist. 

(b) Both P2 and P3 are unstable with one positive and one negative eigenvalues. Interior 
equilibrium exists and is stable. 

(c) Both P2 and P3 are stable with two negative eigenvalues. Interior equilibrium exists 
with one positive and one negative eigenvalue. 

2. If S2, S3 are both in the inferior case, then there are 4 cases. 

(a) Both P 2 and P3 are unstable, one has two positive eigenvalues and the other has one 
positive and one negative eigenvalues. Interior equilibrium does not exist. 

(b) Both P2 and P3 are unstable with two positive eigenvalues. Interior equilibrium exists 
with one positive and one negative eigenvalues. 

(c) Both P 2 and P3 are unstable with one positive and one negative eigenvalues. Interior 
equilibrium exists with two positive eigenvalues. 

3. If S2 is in the inferior case and S3 is in the superior case, then there is only 1 case. P2 
is unstable with two positive eigenvalues and P3 is stable with two negative eigenvalues. 
Interior equilibrium does not exist. 

4- If S2 is in the superior case and S3 is in the inferior case, then there are 2 cases. 

(a) P2 is stable with two negative eigenvalues and P3 is unstable one positive and one 
negative eigenvalues. Interior equilibrium does not exist. 

(b) P 2 is stable with two negative eigenvalues, P3 is unstable with two positive eigenval- 
ues. Interior equilibrium exists with one positive and one negative eigenvalue. 

B. Suppose Pi,P3 exist but not P 2 . Then there are 12 cases. The results are identical to Part 
A above except P 2 and P\ are interchanged in the statements of Part A. In addition, there is 
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an extra case in Part A 4 which we call 4(c): P\ is unstable with one positive and one nega- 
tive eigenvalues, P3 is unstable with two positive eigenvalues. Interior equilibrium does not exist. 

C. Suppose Pi,P2 exist but not P3. Then there are 11 cases. The results are identical to 
Part A above except Pi is changed to P3, P2 is changed to Pi, and P3 is changed to P2 in 
the statements of Part A. In addition, statement 4(a) in Part A is changed to the following: 
Pi is unstable with one positive and one negative eigenvalues, P2 is unstable with two positive 
eigenvalues. Interior equilibrium does not exist. 

Lemma 5.4. Suppose P 1 ,P 2 ,P3 all exist. Then there are 26 cases: 

1. If Si, S2, S3 are all in the inferior case, then there are 4 cases: 

(a) Pi,P2,P% are all unstable and each has one positive and one negative eigenvalues. 
Interior equilibrium exists and has two positive eigenvalues. 

(b) Pi,P 2 ,Pz are all unstable. One of the Pi's has two positive eigenvalues and the rest 
have one positive and one negative eigenvalues. Interior equilibrium does not exist. 

2. If Si is in the superior case and the other two sides are in the inferior case, then there 
are 9 cases. 

(a) Pi is stable with two negative eigenvalues and the other two equilibria are unstable 
with two positive eigenvalues. Interior equilibrium exists with one positive and one 
negative eigenvalues. 

(b) Pi is stable with two negative eigenvalues and the other two equilibria are unstable, 
one has two positive eigenvalues while the other has one positive and one negative 
eigenvalues. Interior equilibrium does not exist. 

3. If Si is in the inferior case and the other two sides are in the superior case, then there 
are 9 cases. 

(a) Pi is unstable with two positive eigenvalues and the other two equilibria are stable 
with two negative eigenvalues. Interior equilibrium exists with one positive and one 
negative eigenvalues. 

(b) Pi is unstable with two positive eigenvalues, one of the other equilibria is unstable 
with one positive and one negative eigenvalues, and the third equilibrium is stable 
with two negative eigenvalues. Interior equilibrium does not exist. 

4- If all three edges are in the superior case, then there are 4 cases. 
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(a) Pi,P 2 ,P 3 are all unstable with one positive and one negative eigenvalues. Interior 
equilibrium exists with two negative eigenvalues. 

(b) One of the boundary equilibria is stable with two negative eigenvalues and the other 
two are unstable with one positive and one negative eigenvalues. Interior equilibrium 
does not exist. 

To prove Lemmas I5.1H5.41 we need to work with the differences of the r^-'s. Let 

«i = r 2 i - r n a 2 = r 31 - r 21 a 3 = r 31 - r u = a Y + a 2 

Pi = r 22 - r 12 (3 2 = r 32 - r 22 (3 3 = r 32 - r 2X = f3 1 + f3 2 (5.2) 

7i = r 23 - n 3 72 = r 33 - r 23 7 3 = r 33 - r i3 = 71 + 72 • 

From Proposition 14.11 the interior equilibrium is given by p* = Ui/ Uj, i — 1, 2, 3 where 

Ui = 7 2 /3i - 7i/? 2 = 73A - 71/^3 , 

U 2 = a 2 7i - "172 = «37i - "i73 , (5.3) 
U 3 = (3 2 ai - (3ia 2 = /3 3 «i - f3ia 3 . 

Let Aij = ru + rjj — 2^-. The eigenvalues of J at the vertex and boundary equilibria are 



Equilibrium 


Coordinates 


First Eigenvalue 


Second Eigenvalue 


Vertex 1 


(0,0) 


-73 


-72 


Vertex 2 


(0,1) 


fa 


-Pi 


Vertex 3 


(1,0) 




a 3 


Pi 






-Ui 




A 23 


A 23 


P2 


/ 73 \ 
\^13 / 


-a 3 7 3 


-u 2 


Ai 3 


Ai 3 


P3 


/ fa -a>i\ 


-Oiifii 


-U 3 


U12' A 12 J 


A12 


A12 



The formulas for the eigenvalues of J at the interior equilibrium are too complex to be 
useful. However, we can determine their signs from the determinant and trace of J which are 

det(J) = plp* 2 p* 3 (Ui + U 2 + U 3 ), (5.4) 

tr(J) = -A 13 (^) 2 - A 23 (^) 2 + (A 12 - A 23 - A 13 )p* 1 p* 2 + A 13 p* 1 + A 23 p* 2 
UiU 2 A l2 + U!U 3 A 13 + U 2 U 3 A 23 
(Ui + U 2 + U 3 f 

We only prove the simplest and the most difficult cases since the rest are similar. 
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Proof of Lemma \5.1\ From Proposition 14. 1[ interior equilibrium exists if and only if all the 
Ui's are of the same sign. Suppose all the C/j's are positive. Then from above, we have 

7iA> < TaA > 72"i < "27i , Pi<*2 < P2OL1 , 

7iR <(-)(-), H(-) <«27i, (-)«a <(-)(-)■ 
Therefore, «2 and 71 are of the same sign. They must both be negative since «2+7i = ^23 — r i2 = 
01 + 02 < 0. But then the above inequalities imply that 71/72 < 0i/02, cti/ct 2 < 71/72, 0i/02 < 
a.x/a.2 which is a contradiction. Therefore, C/j's cannot be all positive. The proof of the fact 
that C/j's cannot be all negative is similar. Hence, interior equilibrium cannot exists. The proof 
of the lemma is complete. 



Proof of Lemma 5.4 We only prove parts 1 and 2 since the proofs of the rest are similar. 
Suppose all three sides are in the inferior case so that Ai 2 , A 13 and A 2 3 are all positive and the 
signs of otijfii, 7^ are 

ai (-), a 2 (?), a 3 (-), 02(~), 7i (?), 72 (+), 7s (+) • 

From (15. 3ft . we have 

C/i = (+)(+) -7i(") = (+)(+) -7i/?3, 
02 = «27i "(-)( + ) = (-)7i -(-)( + ), 
U 3 = (-)(-) - (+)oa = AH -(+)(-)■ 

Since the signs of 02, ^3 and 71 are indeterminate, we construct a table showing the signs of 
C/j's under all possible combinations of the signs of ai,^ 3 and 71. 



Case 


«2 P3 


7i 


C/i 


c/ 2 


03 


(i) 


+ + 


+ 


+ 


+ 


? 


(ii) 


- + 


+ 


+ 


? 


+ 


(hi) 




+ 


+ 


? 


+ 


(iv) 






? 


+ 


+ 


(v) 


+ + 




+ 


+ 


? 


(vi) 


+ - 




? 


+ 


+ 



From the above table, the signs of the eigenvalues of J at the boundary equilibria are 

-01 1 r, f+ ^2 1 n f+ "03 



Recalling Lemma 15.41 Part 1 and referring to the above table, we have 

Cases (i) and (v). If C/3 > 0, then Case (a) happens and if C/3 < 0, then Case (b) happens. 
Cases (ii) and (hi). If C/ 2 > 0, then Case (a) happens and if C/ 2 < 0, then Case (b) happens. 
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Cases iv) and (vi). If U\ > 0, then Case (a) happens and if U\ < 0, then Case (b) happens. 

The proof of Lemma [5.41 Part 1 is complete. To prove Part 2, suppose i — 1 which means that 
Si is in the superior case (r 33 < r 22 < r 23 ) and S 2 , S 3 are in the inferior case (r 13 < r 33 < 
fii, r±2 < r 22 < Tn). Then Ai 2 , Ai 3 are positive and A 23 is negative. The signs of a*, $,7$ are 

ai(-), « 2 (?), a 3 (-), /3 2 (+), /3 3 (+), 71 (+), 72 H, 7s (+) ■ 

Note that U± = — = (— )(+) — (+)(+) < an d the signs of the eigenvalues of J at 
the boundary equilibria are 

Mp^}' Mp^}- 

Therefore Pi is always stable with two negative eigenvalues. Suppose a 2 > 0, then U3 = 
fiiot-x — fi\ot-i < and P 3 {+, +}. If £/ 2 > 0, then P 2 {+, — } and Case (b) happens. If U 2 < 0, 
then P 2 {+, +} and Case (a) happens. Suppose a 2 < 0, then U 2 = a 2 ^\ — «i7 2 < and 
P2 {+,+}• If ^3 > 0, then P 3 {+,-} and Case (b) happens. If U 3 < 0, then P 3 {+, +} and 
Case (a) happens. The proof of Lemma [5.41 Part 2 is complete. 

The 69 cases in the above four lemmas are summarized in Tables [TJ{3] and the 14 patterns 
are separated by double lines in the tables. 
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T 

Lemma 


(0,0) 


(0,1) 


(1,0) 


Pi 


Pi 


1 ) 

P3 


T 4- 

Interior 


3.1 


+ + 


+ 




X X 


X X 


X X 


X X 


O O 1 I n\ 

o.Z.iyt ) 


1 

— + 


1 

— + 




X X 


1 1 

+ + 


X X 


X X 


^ 9 9M 


1 

— 


1 

— \ 




1 1 


X X 


X X 


X X 


O.Z.Zl 666 J 


1 1 

~T~ ~T 


1 1 




1 

-T 


X X 


X X 


X X 


^ ^ R ztf/^ 


1 1 


1 

\ — 




1 

— 


X X 


1 1 


X X 


O.O.Ly.ft^U J 


1 

— T 


1 1 
\ \ 




1 


1 1 


X X 


X X 


Q Q D 1 

o.o.ij.i^aj 


1 1 


1 1 


1 

+ — 




X X 


1 


X X 




1 1 


1 1 
\ \ 


1 

— T 




_l_ 

— \ 


X X 


X X 




1 1 


1 

1 — 


1 

— T 




_l_ 

— \ 


1 1 


X X 


^ zt 3 I'M 


1 

— ~~r 


1 1 
I \ 


1 

~r — 




1 1 
T \ 


1 

— ~r 


X X 


^ 4 zUM 


i i 


1 1 
I \ 


i i 




1 

— -r 


1 

— ~r 


X X 


6.S.C.l{a) 


+ + 


+ + 


+ 


+ 




X X 


X X 


3.3.C.3 


+ 


+ 


+ 


+ + 




X X 


X X 


oA.6{0) 


1 1 

+ + 


1 

+ — 


1 

— + 


1 

— + 




1 1 

+ + 


X X 


^ zt 3 I'M 


l 

~r — 


1 

— ~r 


i i 


1 1 




1 

— ~r 


X X 


^ zt zt(M 


I i 
\ \ 


I i 
\ \ 


I i 
\ \ 


I 




1 


X X 


O.O./i. 1 J 


I i 


I 

— 


I i 


X X 




1 


X X 


O.Z. 1 I 66 I 


1 1 


1 


i 

— "T 


X X 




X X 


X X 




+ + 


+ 


+ 


X X 


X X 




X X 


3. 3.Al(a) 


+ + 


+ 


+ + 


X X 


+ 




X X 


3. 3. A3 


+ 


+ 


+ 


X X 


+ + 




X X 


o.o.-D.l^aj 


1 1 

+ + 


1 1 

+ + 


1 

+ — 


1 

— + 


X X 




X X 




1 

~T~ — 


1 

— t 


1 

~T~ — 


1 1 

~r ~r 


X X 




X X 


^ zt 3 I'M 


1 

— T 


1 i 


1 

— 


1 

— \ 


1 1 
\ \ 




X X 


^ Zt ^ I'M 


1 

+ — 


1 

— 


1 1 


1 1 


1 

— \ 




X X 


3 zt A(h\ 


_|_ _|_ 
\ \ 


_|_ _|_ 


_|_ _|_ 


_|_ 


_L 




v V 


3.3.^4.1(6) 


+ + 


+ 


+ + 


X X 


+ 


+ 




3.3.5.1(6) 


+ + 


+ + 


+ 


+ 


X X 


+ 




3.3.(7.1(6) 


+ + 


+ + 


+ 


+ 


+ 


X X 




3.4.4(a) 


+ + 


+ + 


+ + 


+ 


+ 


+ 





Table 1: One stable equilibrium: 30 cases 5 patterns. +, — are the signs of the eigenvalues of 
J at the corresponding equilibrium and x means equilibrium does not exist. 
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T 

Lemma 


(0,0) 


(0,1) 


1 1 n\ 

(1,0) 


7 } 


Pi 


P3 


T 4 

Interior 


6.2.6(h) 


+ + 






X X 


X X 


+ + 


+ 


6. A.oyill) 


+ + 






X X 


X X 


+ 


X X 


OO ZD O / ^ \ 

d.d.i3.z(aj 


+ 






+ 


X X 


+ + 


X X 


d.o.-D.2(aJ 


+ 






+ + 


X X 


+ 


X X 


6.6. n. 2(b) 


+ 






+ + 


X X 


+ + 


+ 


O.O.IJ .Z[C) 


1 

+ 






1 


X X 


1 

+ — 


1 1 

+ + 


3 3 4 9fM 


1 

— \ 






X X 


1 

\ — 


1 1 


X X 


o.o./i.zi a 1 


1 

— ~r 






X X 


1 1 
~r ~r 


1 

~r — 


X X 


Q Q 4 9 I'M 


1 

— T 






X X 


1 1 
T ~T~ 


1 1 


1 

~r — 


S 3 4 2fr) 


_L 






Y Y 


4 _ 


4 — 


_|_ 4 






1 

— ~r 




1 

4 — 


1 1 

4 4 


X X 


X X 


q q r» 9^ 




— 4 




1 1 
\ \ 


1 

\ — 


X X 


X X 


q q of/,'! 




1 

— \ 




1 1 

4 4 


1 1 


X X 


1 

\ — 


O.O.Ly .Zl J 




1 

— \ 




4 — 


1 

4 — 


X X 


1 1 


6.2.2(h) 


+ + 


+ + 






X X 


X X 


+ 


6.6.BA(a) 


+ + 


+ 






X X 


+ 


X X 


q q n A (h\ 

O.O. ID .^(O) 


1 1 

-r -r 


1 

+ 






X X 


1 1 


1 

+ — 




1 

— ~r 


1 
1 






1 1 

4 4 


1 1 
1 1 


1 

4 — 


^4 9 I'M 


1 


1 
1 






1 1 

4 4 


1 

1 — 


X X 


^ 4 9i'M 


1 


1 

4 






4 — 


1 1 
\ \ 


X X 


'i. ^ H A(K\ 

O.O.L/.^O ) 


1 


1 1 

~T~ 4 






1 1 

4 4 


X X 


1 

4 — 


O A A ! n\ 

o.o.AA(a ) 


1 1 

+ + 




1 


X X 




1 

+ — 


X X 


q q 4 A(h\ 


1 1 

4 4 




1 

4 


X X 




1 1 

4 4 


1 

4 — 


^ 4 0( n\ 

O.Q.Z, 1 (X 1 


1 




1 

— \ 


1 1 
\ \ 




1 1 


1 

— 


3 4 2 CM 


-|_ 




— 4. 


4 4 




4 _ 


X X 


3.4.2(6) 


■ — 




— + 


+ — 




+ + 


X X 


3 4 2M 




_ -L 


4 _ 


4 4 


4 4 




4 _ 


3.4.2(6) 




— 4- 


4- — 


4 4 


+ — 




X X 


3.4.2(6) 




+ 


4 


4 


+ + 




X X 


3.3.C.l(c) 


4- 4- 


+ + 


— + 






X X 


+ — 


3.4.3(a) 


+ + 


+ — 


— + 






+ + 


+ — 


3.3.5.1(c) 


+ + 


+ + 


+ 




X X 




+ 


3.4.3(a) 


+ 


+ + 


+ 




+ + 




+ 


3.3.Al(c) 


+ + 


+ 


+ + 


X X 






+ 


3.4.3(a) 


+ 


+ 


+ + 


+ + 






+ 



Table 2: Two stable equilibria: 35 cases 8 patterns. +, — are the signs of the eigenvalues of J 
at the corresponding equilibrium and x means equilibrium does not exist. 
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Lemma 


(0,0) 


(M) 


1 1 rA 


7 ) 


7 ) 


] ) 


Interior 


3.4.1(a) 








+ 


+ 


+ 


+ + 


3.4.1(6) 








+ + 


+ 


+ 


X X 


3.4.1(6) 








+ 


+ + 


+ 


X X 


3.4.1(6) 








+ 


+ 


+ + 


X X 



Table 3: Four cases 1 pattern with three stable equilibria. +, — are the signs of the eigenvalues 
of J at the corresponding equilibrium and x means equilibrium does not exist. 

Knowing the patterns is not enough to predict the asymptotic behavior of solutions of (15. II) . 
We also need to know the separatrices for each pattern. Separatrices are curves which divided 
A into separate regions each containing a stable equilibrium. If the initial condition lies in 
one of these regions, then the solutions of (15. ip will converge to the stable equilibrium in that 
region as t — > oo. The 14 patterns together with their separatrices are shown in pictures in 
the following theorem. In these pictures, big dots and small dots represent stable and unstable 
equilibria, respectively. Arrows indicate the directions of flow of the solutions. 

Theorem 5.5. For the three-allele case without complete dominance, 14 patterns may occur. 

1. There are 5 patterns with one stable equilibrium. They are {(1,0)}, {-Pi}, {P2}, {-£3}; 
and the interior equilibrium, respectively. There is no separatrix. (See Tabled) 

2. There are 8 patterns with two stable equilibria. (See Tabled) They are 

(a) {(0, 1), (1, 0)} : The separatrix may behave in 10 different ways. See Fig. [7] and\E 




3.3.A.2(C) 3.3.B.2(a) 3.3.B.2 (a) 3.3.B.2(b) 3.3.B.2(C) 

Figure 2: {(0, 1), (1, 0)} 
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(b) {(0,0), (1,0)} : The separatrix may behave in 4 different ways. See Fig. [3J 




3.3.C.2 (a) 



3.3. C. 2 (a) 



3.3. C. 2(b) 



3.3.C.2 (c) 



Figure 3: {(0, 0), (1, 0)} 
(c) {(1,0), Pi} : The separatrix may behave in 7 different ways. See Fig. ^ and{5[ 




3.2.2 (ii) 















C— i (: 





3.3.B.4(a) 3.3.B.4(b) 3.3.C.4(b) 

Figure 4: {(1,0), A} 




3.4.2 (b) 



3.4.2(a) 3.4.2(b) 

Figure 5: {(1,0), A} 
(d) {(0, l),^} : The separatrix may behave in 5 different ways. See Fig. 




3.3.A.4 (a; 



3.3.A.4(b) 3.4.2(a) 3.4.2(b) 

Figure 6: {(0,1), P 2 } 



3.4.2 (b) 
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(e) {(0,0), P3} : The separatrix may behave in 3 different ways. See Fig. [?| 




3.4.2(a) 3.4.2(b) 3.4.2(b) 

Figure 7: {(0,0), P 3 } 
(f) {P11P2} '■ The separatrix may behave in 2 different ways. See Fig. [21 




3.3.C.1 (c) 3.4.3(a) 

Figure 8: {P U P 2 } 

(g) {Pi,Ps} : The separatrix may behave in 2 different ways. See Fig. 




3.3. B. 1 (c) 3.4.3(a) 

Figure 9: {P U P 3 } 

(h) {P2,Pz\ ■ The separatrix may behave in 2 different ways. See Fig. UU 




3.3.A.1 (c) 3.4.3(a) 

Figure 10: {P 2 , P 3 } 
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3. There is only one pattern with three stable equilibria which must be the three vertices. (See 
Tabled) The separatrix may behave in 4 different ways. See Fig. Uli 




3.4.1(a) 3.4.1(b) 3.4.1(b) 3.4.1(b) 

Figure 11: {(1, 0), (0, 1), (0, 0)} 



5.2 Complete Dominance Case 

In this subsection, we consider the case of complete dominance on at least one side of A. There 
are altogether 26 cases to consider and the number of patterns is greatly reduced in each case. 

(i) Complete dominance on one side. 

There are 6 ways to assume complete dominance on one side. 

(a) Suppose 5*3 is completely dominant (r i2 = rn or r i2 = r 22 ). The dynamics of ( 15. ip on 
5*3 is the same as S3 being in the intermediate case. The dynamics on the other two edges will 
not be affected if S3 is changed from intermediate case to complete dominance case. Therefore, 
to find all the patterns, we go through every case of S3 being in the intermediate case in Tables 
1-3 and examine how the signs of U-s will change when the condition r 22 < ri 2 < rn is replaced 
with ri2 = r\i or ri2 = r 22 . There are 17 such cases which generate 7 patterns: 

{(1,0)}, {P 1 }, {P 2 }, {!}, {(0,0), (1,0)}, {(1,0), Pi}, {Pi,P 2 }. 

In the above X means interior equilibrium. We observe that if ri 2 = rn, the pattern {Pi,P 2 } 
(Lemma 3.3.0.1(c) in Table 2) did not show up in our simulations. We now prove that this 
pattern cannot occur. From Table 2, sixth line from the bottom, we see that Si, S 2 are in the 
superior case (r 33 < r 22 < r 23 , r 33 < r n < r 13 ). This and the fact r 12 = rn imply that the signs 
of the Greek alphabets defined in ( 15. 2 p are 

«i(0), a 2 (+), a 3 (+), ft (-), &(+), &(?), 7i(?), 7 2 (-), 7s H ■ 

The signs of the eigenvalues of J at Pi, P 2 are Pi{— , — r 1 } and P 2 {— , = z z }- If Pi, P 2 are both 
stable, we have U\ < and U 2 < 0. From (|5T3|) . U x = <y 2 /3i - 7i/3 2 = (—)(—) - 7i(+) and 
U2 = «27i — «i72 = (+)7i which imply that Ui,U 2 cannot both be negative. Therefore, the 
pattern {Pi, P 2 } cannot occur. If ri 2 = r 22 , then among the above 7 patterns, only the pattern 
{X} (Lemma 3.3.C.l(b) in Table 1) cannot occur. The proof of this fact is similar and will be 
omitted. 
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(b) Suppose 5*2 is completely dominant (ri 3 = m or r i3 = r 33 ). We repeat the process above 
and find that there are 19 cases of S 2 being in the intermediate case which generate 7 patterns: 

{(1,0)}, {PJ, {P 3 }, {X}, {(0,1), (1,0)}, {(1,0), Px}, {Pi,P 3 }. 

However, if r 13 = ru, the pattern {P 1; P 3 } (Lemma 3. 3. B. 1(c) in table 2) cannot occur. If 
r i3 = r 33) the pattern {X} (Lemma 3. 3. B. 1(b) in table 1) cannot occur. The proof of these fact 
will be omitted. 

(c) Suppose Si is completely dominant (r 23 = r 22 or r 23 = r 33 ). We repeat the process above 
and find that there are 17 cases of Si being in the intermediate case which generate 7 patterns: 

{(1,0)}, {P 2 }, {P 3 }, {X}, {(0,1), (1,0)}, {(0,1), P 2 }, {P 2 ,P 3 }. 

However, if r 23 = r 22 , the pattern {P 2 ,P 3 } (Lemma 3. 3. A. 1(c) in table 2) cannot occur and if 
r 23 = r 33; the pattern {X} (Lemma 3. 3. A. 1(b) in table 1) cannot occur. The proof of these 
facts will also be omitted. 

(ii) Complete dominance on two sides. 

There are 12 ways to assume complete dominance on two sides. The complete list is given 
below. They are all confirmed by computer simulations. 

(a) S3 and S2 are completely dominant: {(1,0)}, {(1,0), Pi}. 

(b) S3 and Si are completely dominant: {(1,0)}, {P 2 } • 

(c) S 2 and Si are completely dominant: {(1,0)}, {P 3 }, {(0, 1), (1,0)} . 

(iii) Complete dominance on all sides. 

There are 8 ways to assume complete dominance on all sides. They all generate the same 
pattern {(1, 0)}. 

6 Higher Number of Alleles 

For k alleles, k > 4, there are too many possible patterns that we cannot prove their existence 
like we did for the three-allele model in Section [57T1 We therefore resort to computer simulations 
to verify that certain patterns exist. We also develop some rules to show that certain patterns 
do not exist. For example, according to Proposition 14.51 we have 

Rule 1. For any k, if the interior equilibrium is stable, then no other equilibrium can be stable. 

We say that three edge equilibria are coplanar if the edges they lie on form a triangle. From 
the results of the three-allele model, we have the following 

Rule 2. The maximum number of coexisting stable equilibria in A is 3 and they must be the 
three vertices. Hence, stable coplanar edge equilibria is not possible. 

Rule 3. If the existing stable equilibria are all vertices, then they must include the vertex (1,0). 

Rule 3 is a consequence of condition ( II. 2p . We conjecture that it is valid for any k. 
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6.1 Four-allele Model 



The invariant set A defined by ( 12.31) is a tetrahedron. It contains 15 possible equilibria including 
4 vertices, 6 edge equilibria, 4 face equilibria, and one interior equilibrium. From the results of 
the three allele case, it is easy to see that the maximum number of coexisting stable equilibria 
is 4 which we state below as rule 4. We first construct a table of all 64 possible patterns and 
eliminate those that cannot exist using rules 1, 2 and 6 below. For example, the pattern 3v + / 
(three stable vertices and one stable face equilibrium) is not possible because such a pattern 
would imply a three-allele model with two stable vertices and a stable edge equilibrium which 
violates rule 2. We also eliminate the patterns 3/, e + 2/ and 3e + / which are shown to be 
not possible in [16] . The remaining patterns are shown to exist by computer simulations and 
are summarized in Table H] below. In the table, N s denotes the number of coexisting stable 
equilibria and N p denotes the number of patterns for the type indicated. If we add up all the 
numbers in the N p column, we see that the four-allele model has 117 patterns. The last column 
in the table explains how the numbers in the column N p are obtained. For example, according 
to Table HI there are 12 patterns with one vertex equilibrium and one edge equilibrium; that is, 
If + le. There are six edges in a tetrahedron and we can pair anyone (hence (Tj = 6 choices) 
with any one of the four vertices that does not lie in the edge. Since a tetrahedron has four 
vertices and each edge connects two vertices, the number of choices here is ( 4 ^ 2 ) = 2. The 
notation (™) means m choose n which equals m\/{n\{m — n)\). Therefore, there are 6 x 2 = 12 
such patterns. We develop the following rules which will be used in the study of the five-allele 
model. The proof of rule 6 is rather involved and will not be presented here. Rule 5 follows 
from looking at Table 4. 

Rule 4- The maximum number of coexisting stable equilibria in A is 4. 

Rule 5. If A contains three or more stable equilibria, then none can be on a face except we 
can have a face with a stable equilibrium and two stable edge equilibria not on the face. 

Rule 6. If the stable equilibria are all vertices, then they must include the vertex (1, 0, 0). 
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■Lis 


pattern type 




J^JvUlclllcLLlUllo 


V 


e 


/ 


t 




1 








1 


must be (1, 0, 0). 


1 




1 






6 


any one of the six edges. 






1 




4 


any one of the four faces. 










1 


1 


the interior equilibrium. 




2 








3 


any two vertices one of which must be (1, 0, 0). 






2 






15 


any two edges: (£) = 15. 








2 




6 


any two faces: (*) = 6- 


2 


1 


1 






12 


pair any edge with a vertex not in the edge: x ( 4 ^) = 12. 




1 




1 




4 


pair any face with the one vertex not in the face. 






1 


1 




12 


pair any face with an edge not in the face: (*) x ( b ^ a ) = 12. 




3 








3 


three vertices one of which must be (1, 0, 0): C*^ 1 ) — 3. 






3 






16 


any three edges not coplanar: — 4 = 16. 


2 


1 






6 


pair any edge with the two vertices not in the edge. 




1 


2 






12 


pair any vertex with two edges not containing the vertex: 
(i) x (2) = Note two ends of an edge are vertices. 






2 


1 




12 


any face with two of the three edges not from the face. 


4 


4 








1 


all four vertices. 




4 






3 


any four edges without forming a triangle: — 4 x 3 = 3. 



Table 4: Four-allele model with 117 patterns. In the table v denotes vertices, e denotes edge 
equilibria, / denotes face equilibria, and t denotes tetrahedral equilibria. 



6.2 Five-allele Model 

The set A contains 31 equilibria including 5 vertices, 10 edge equilibria, 10 face equilibria, 5 
tetrahedral equilibria and 1 interior equilibrium denoted by %. There are at least 2351 patterns 
shown in the tables below. There are a few patterns which we cannot show nonexistence nor 
found them in our simulations. They are discussed at the end of this subsection. 
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pattern type 


N 


H \'T"i Ian atinnc 
J^JvUlcLllcLLlUllo 


V 


e 


/ 


t 


i 


1 


1 










1 


must be (1,0,0,0). 




1 








10 


any one of the ten edges. 






1 






10 


any one of the ten faces. 








1 




5 


any one of the five tetrahedrons. 










1 


1 


the interior equilibrium. 


2 


2 










4 


one of the two vertices must be (1, 0, 0, 0). 




2 








45 


any two edges: ( ") = 45. 






2 






45 


any two faces: ( 2 ) =45. 








2 




10 


any two tetrahedrons: ( 2 ) = 10. 


1 


1 








30 


any edge and a vertex not in the edge: 10 x 3 = 30. 


1 




1 






20 


any face and a vertex not in the face: 10 x 2 = 20. 


1 






1 




5 


tetrahedron and the vertex not in the tetrahedron. 




1 


1 






70 


any face with an edge not in the face: 10 x 7 = 70. 




1 




1 




20 


any tetrahedron and an edge not in the tetrahedron: 
(?) x CV) = 20. 






1 


1 




30 


any tetrahedron and a face not in the tetrahedron: 

(?) x rr 4 ) = so. 


3 


3 










6 


one of the three vertices must be (1, 0, 0, 0): ( 5 9 1 ) = 6. 




3 








110 


any three edges not coplanar: (^ u ) — 10 = 110. 






3 






100 


any three faces not in a common tetrahedron: 
(-) - ffl x 5 = 100. 


2 


1 








30 


pair any edge with two vertices not in the edge: 
(T) x (V) = 30. 


2 




1 






10 


pair any face with the two vertices not in the face. 


1 


2 








75 


any vertex with two edges not containing the vertex: 
(?) x (V) = 75. 




2 


1 






210 


pair any face with two edges not in the face: 
10 x ( 10 T 3 ) = 210. 




2 




1 




30 


pair any tetrahedron with any two edges not in the 
tetrahedron: Q) x ( 10 T 6 ) = 30. 






2 


1 




75 


pair any tetrahedron with any two faces not in the 
tetrahedron: (J) x ( 10 " 4 ) = 75. 




1 


2 






180 


any edge with two faces not containing the edge and 
not in a common tetrahedron with the edge (an edge 
is shared by three faces and three tetrahedrons): 
10 x (( 10 ~ 3 ) - 3) = 180. 



Table 5: Five-allele model with at least 2351 patterns. 
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N s 


pattern type 


N p 


Explanations 


V 


e 


/ 


t 


% 




1 




2 






30 


pair any vertex with two faces not containing the vertex 


3 














(a vertex is shared by six faces): (J x ( 10 ^ 6 ) = 30. 




1 


1 


1 






60 


a face with a vertex not in the face and an edge neither 
in the face nor containing the vertex: 10 x 2 x 3 = 60. 






1 


1 


1 




60 


any tetrahedron with a face not in the tetrahedron plus 
an edge neither in the tetrahedron nor in the face: 
5 x 6 x 2 = 60. 




4 










4 


one of the four vertices must be (1, 0, 0, 0): ( t> ^ i ) = 4. 






4 








140 


any four edges without forming a triangle: 

m - m x m = mo. 








4 






75 


see A below. 




2 


2 








30 


pair any two vertices with any two edges not containing 


4 














these vertices: (I) x (fj = 30. 






2 


2 






150 


see B below. 




3 


1 








10 


pair any edge with the three vertices not in the edge. 




1 


3 








80 


any three edges without forming a triangle from a 
tetrahedron (see pattern 3e in the four- allele model) 
and the vertex not in the tetrahedron: 16 x 5 = 80. 






3 


1 






60 


see C below. 






3 




1 




20 


pair any tetrahedron with any three edges not in the 
tetrahedron: Q) x Q = 20. 






1 


3 






120 


see D below. 








3 


1 




60 


see E below. 




1 


2 


1 






60 


pick two edges and one face from a tetrahedron and 
choose the vertex not in the tetrahedron: 5 x 12 = 60. 
(see pattern v + 2e in four-allele model). 




5 










1 


all five vertices. 






5 








72 


see F below. 


5 






5 






12 


see G below. 


1 


4 








15 


any four edges without forming a triangle from a 
tetrahedron (see pattern 4e in the four-allele model) 
and the vertex not in the tetrahedron: 3 x 5 = 15. 






4 


1 






90 


type face {123} with edges {14} {15} {24} {25}: 10 x 3. 
type face {123} with edges {14} {35} {24} {45}: 10 x 6. 






3 


2 






60 


see H below. 


6 




6 








10 


choose the six edges in any two tetrahedrons not on 
the common face of the two tetrahedrons: = 10. 



Table 6: Five-allele model with at least 2351 patterns (continued). 
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To understand the explanation column of the above table, take the first pattern If + 2f in 
Table |6j There are (J vertices and suppose a vertex, say vertex 1, is chosen. Then among the 
10 faces of A, (t,) = 6 of them contains vertex 1. Therefore, 10 — 6 faces do not contain vertex 1 
and there are ( 10 2" 6 ) = 6 ways to choose two of them. This explains why N p — 5 x 6 = 30. The 
explanations of some the patterns in Table [6] are too complicated to fit into the space provided 
in the table and are explained below. 

A. For the 4/ pattern, we observe that each face is shared by two tetrahedrons so there are 
actually 8 faces if we take the tetrahedrons into account. According to Rule 5, each tetrahedron 
can contain at most 2 stable face equilibria. The 8 faces may be put into the 5 tetrahedrons in 
two ways: 2 + 2 + 2 + 2 + or 2 + 2 + 2 + 1 + 1. Consider the first way. There are Q x 3 = 15 
patterns. The number 3 may be explained as follows. Suppose we label the vertices 1 through 
5 and we eliminate the tetrahedron {2345}. Consider one of the remaining four tetrahedrons, 
say {1234}. There are (2) = 3 ways to choose 2 faces from 3 because the face {234}, being on 
{2345}, is not eligible. But once we choose two of the faces, the other two faces on each of the 
remaining 3 tetrahedrons are fixed. This explains the 3 in the formula. For the second way, 
there are x 6 = 60 patterns so altogether we have 15 + 60 = 75 4/ patterns. 

B. For the 2e + 2f pattern, it is easy to see that the two faces chosen must have one 
common edge (10 x g) = 30 ways) or one common vertex (5 x Q/2 = 15 ways because of 
symmetry). In each case, there are 6 ways to choose the two stable edge equilibria in order not 
to violate Rule 5. Thus, 30 x 6 + 15 x 6 = 270 patterns are expected. However, the pattern 
with faces {123}, {134} and edges {15}, {25} is not show up in our simulations and we could 
not find it in any literature. There are 30 x 4 = 120 such patterns. Therefore, there are at least 
150 and at most 270 2e + 2f patterns. 

C. For the 3e + If pattern, first pick one of the 10 faces, then choose 3 edges such that 
they are not coplanar (Rule 2), none of them is from the face chosen (Rule 1), and they and the 
face are not in a common tetrahedron (Rule 5). Accordingly, there are 10 x (Q - 3-2) = 300 
patterns expected. But only patterns of the type face {123} with edges {14}, {25}, {45} 
(10x3x2 = 60 patterns) show up in our simulations, whereas the patterns of the type face 
{123} with edges {14}, {34}, {45} (60 patterns), of the type face {123} with edges {14}, {34}, 
{15} (60 patterns), and of the type face {123} with edges {14}, {34}, {25} (60 patterns) do not 
show up in our simulations. Therefore, there are at least 60 and at most 300 3e+lf patterns. 

D. For the le + 3/ pattern, there are two types of patterns. For type 1, first pick one of 
the 10 faces, say, {345} as face 1, then face 2 is determined by vertex {1} and any edge of face 

1, say, {134}, face 3 is determined by vertex {2} and any edge of face 1 but not used by face 

2, say, {245}. The stable edge equilibrium is on {12}. There are 10 x 3 x 2 = 60 patterns of 
type 1. For type 2, first pick two of the 10 faces with a common edge (30 ways), say, {134} and 
{234}, then the third face is {125} decided by the left vertex {5} and the two vertices not on 
the common edge of the first two faces. Lastly, the stable edge equilibrium could be either on 
{35} or {45}. There are 30 x 2 = 60 patterns of type 2. 
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E. For the 3/ + It pattern, first pick any tetrahedron, say, {1234}. To decide the three 
faces, we pick three edges from {1234} which are not coplanar and not all from the same vertex, 
say, {12}, {13}, {34}, then the three faces are determined by vertex {5} and these three edges 
respectively, namely, {125}, {135}, {345}. There are total 5 x — 4 — 4) = 60 patterns. 

F. For the 5e pattern, we observe that there are 252 ways to choose 5 edges from the 10 
edges but we must subtract 150 cases of one coplanar edges and 30 cases of two coplanar edges 
which will result in only 72 5e patterns. To see that there are 150 one coplanar edges, suppose 
the coplanar edges are the sides of the triangle {345} (we have 10 such triangles). Then the 
remaining two edges can be chosen two ways (so that we don't have two coplanar edges). One 
way is to choose the edge {12} and then the fifth edge may be chosen from any of the six 
edges {13}, {14}, {15}, {23}, {24}, {25}. A second way is to choose one edge from the group 
{13}, {14}, {15} and another edge from the group {23}, {24}, {25} with a total of 9 choices. 
Hence, there are 6 + 9 = 15 ways to choose the other two edges besides the sides of the triangle 
{345}. Therefore, there are 150 cases of one coplanar edges. The argument for the 30 two 
coplanar edges is similar and will be omitted. 

G. For the 5/ pattern, each face is shared by two tetrahedrons so there are actually 10 
faces if we take the tetrahedrons into account and each tetrahedron has exactly two stable face 
equilibria. Choose two faces (6 ways) from the first tetrahedron {1234}, say, {123} and {124}. 
Then in the second tetrahedron {1235}, we choose one more face either {235} or {135}. The 
two faces chosen in the rest tetrahedrons are fixed now. Thus, there are 6 x 2 = 12 ways. 

H. For the 3e + 2f pattern, choose any two faces with a common edge (30 ways), say, 
{234} and {345}, then choose three edges out of {12}, {13}, {14}, {15} such that they are not 
in a common tetrahedron with either {234} or {345} by Rule 5. Thus, there are 30 x 2 = 60 
ways. 

We cannot find the following patterns in our simulations nor can we prove their nonexistence: 
If + 2t, 2f + 2t, le + 2f + It, 4/ + It, some types of 2e + 2f (see B above), and some types 
of 3e + / (see C above). From the tables we observe the following rules which may be used to 
study the six-allele model: 

Rule 7. The maximum number of coexisting stable equilibria for the five-allele model is 6 with 
all stable equilibria on the edges. 

Rule 8. If there are five stable equilibria, none of them can be in the interior of a tetrahedron. 

Rule 10. If the stable equilibria are all vertices in five-allele case, then one of them must be 
(1,0,0,0). 

6.3 Computer Simulations 

In this section we briefly describe how one can perform computer simulations to help find 
patterns for higher number of alleles. One possible idea which we adopted is the following. First 
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write a computer program that randomly generates a fitness matrix R that satisfies condition 
(11. 2p . Then use the method described in Remark 14.31 to compute and count the number of 
negative eigenvalues at each existing equilibrium. Note that for k alleles, there are a maximum 
possible 2 k — 1 equilibria. Write the results of the computer program to a text file. Each line in 
the text file should contain 2 k — 1 integers representing the number of negative eigenvalues of 
the Jacobian matrix evaluated at that particular equilibrium. A special symbol should be used 
in case the equilibrium does not exist. Run a sorting algorithm on the text file that will turn 
each line into a pattern (see example below). One needs to repeat this process several times, 
each time with increasing number of simulations until no new patterns are found. 

As an example, suppose we want to find all the patterns for the three-allele model. Then 
a line in our output text file may look like —1 — 2 — 2 + + 9 — 1 + 9, which means that the 
three vertices have 1,2 and 2 negative eigenvalues, respectively, Pi,P$ have and 1 negative 
eigenvalues, respectively, and P2 and the interior equilibrium do not exist for this round of 
simulation (9 means the corresponding equilibrium does not exist). Of course another fitness 
matrix R will generate a different line of numbers. The sorting algorithm will turn this line into 
the pattern # — 2 — 2##### and write it to a file called "filename. 2" which contains all 
distinct patterns with two stable equilibria, "filename" is a user supplied name and the sorting 
algorithm also removes duplicate patterns. At the end, the sorting algorithm produces three 
files: "filename. 1", "filename. 2" and "filename. 3". "filename. 1" contains 30 distinct patterns 
with one stable equilibrium (see Tabled]), "filename. 2" contains 35 distinct patterns with two 
stable equilibria (see Table [2]) and "filename. 3" contains 4 distinct patterns with three stable 
equilibria (see Table [3]). 

Some patterns are difficult to find if k is large. For example, for the five-allele case, we need 
to simulate over 40 million times until no new patterns are found and yet we are not sure if we 
found them all. For the six-allele case, we used 20 processors and simulated over 400 million 
times and found over 60, 000 distinct patterns. Of course, not finding a pattern does not mean 
that pattern does not exist unless its absence can be proved theoretically. We believe that the 
number of patterns increases exponentially with k. 

7 Conclusion 

In this paper, we list all patterns for the three- and four-allele selection models and almost 
all patterns for the five-allele selection model for all fitness matrices satisfying condition (II. 2ft . 
A pattern is simply the number and locations in the invariant set A of all coexisting stable 
equilibria. In our investigation, we do not assume the truth of the conjecture of Cannings 
and Vickers that any subset of a pattern is also a pattern [T6l [5]. Knowing the patterns is 
important because it tells us the possible long-term genetic makeup of the population; that 
is, which alleles will survive and which will disappear under selection forces in the long run. 
For the three-allele model, we prove our results analytically and show the separatrices for each 
pattern so that we know the asymptotic behavior of the solutions of the mathematical model 
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(11. lp as t — > oo. We also consider the case of complete dominance for the three- allele model. 
Through the use of computer simulations and the rules we developed, we show that there are 
117 patterns for the four-allele model and at least 2351 patterns for the five-allele model. There 
are a few patterns in the five-allele model that did not show up in our computer simulations 
but we are not able to show that they do not exist. For the six- allele model, parallel computer 
simulations indicate that there are over 60, 000 patterns. We conjecture that the number of 
patterns grow exponentially as the number of alleles increases. In this paper, we also prove 
several interesting mathematical results in Section H] some of which we believe are new. 
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